Tails of probability density for sums of random independent variables 
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The exact expression for the probability density p„ (x) for sums of a. finite number A'^ of random independent 
terms is obtained. It is shown that the very tail of p„ (x) has a Gaussian form if and only if all the random terms 
are distributed according to the Gauss Law. In all other cases the tail for Pj^ (x) differs from the Gaussian. If the 
variances of random terms diverge the non-Gaussian tail is related to a Levy distribution for p„ (x). However, 
the tail is not Gaussian even if the variances are finite. In the latter case p„ (x) has two different asymptotics. 
At small and moderate values of x the distribution is Gaussian. At large x the non-Gaussian tail arises. The 
crossover between the two asymptotics occurs at x proportional to A'^. For this reason the non-Gaussian tail 
exists at finite N only. In the limit TV tends to infinity the origin of the tail is shifted to infinity, i. e., the tail 
vanishes. Depending on the particular type of the distribution of the random terms the non-Gaussian tail may 
decay either slower than the Gaussian, or faster than it. A number of particular examples is discussed in detail. 
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It is well known that if a random quantity x is a sum of a 
number of independent random variables ^ 



N 



(1) 



the corresponding probability density p{x) is described by a 
Gaussian distribution (GD), provided the conditions of the 
Central Limit Theorem (CLT) of Probability Theory [|l]] are 
fulfilled. However, in reality while GD fits well the profile of 
p{x) at small and moderate vales of x the very tail of p{x) 
often deviates from the Gauss Law. Most of the observed de- 
viations are related to the so-called heavy tails which 
exhibit a decay slower than the Gaussian (usually the heavy 
tails decay as a certain power of x). It should be stressed that 
despite the heavy tails do not influence much the average char- 
acteristics of X, such as, e.g., differen moments of it, large 
fluctuations of x are described by the tails entirely. There- 
fore the tails Qfp{x) play an important role in those problems, 
where rare but large fluctuations are of special interest, for ex- 
ample for temperature fluctuations in a nuclear reactor, where 
too big fluctuations may result in damage of the reactor. 

The heavy tails have been observed in a wide diversity of 
phenomena ranging from the statistics of solar flares [Ij] to the 
one of casualties of wars PI and from market price fluctua- 
tions [Ql to fluid dynamics It provides grounds to suppose 
that the phenomenon should have a general cause, which is 
not related to a particular realization of a random sum in a 
given problem. However, to the best of my knowledge such a 
cause has not been revealed yet. 

The present contribution is an attempt to fill the gap. It is 
shown that occuiTence of non-Gaussian tails is quite a com- 
mon case, which does not require any special conditions to be 
realized. Visa versa, special (and very strict) conditions are 
required for realization of Gaussian tails. It is shown also that 
depending on the distribution of random ^ the tails of p{x) 
could be either heavy, or light. In the latter case the tails de- 
cay faster than the Gaussian. Examples of tails of both types 
are considered in detail. 



For the sake of simplicity in what follows sums of inde- 
pendently and identically distributed random variables with 
zero mean values are considered. Extension of the approach 
to more general cases is straightforward. 

First of all notice that, of course, nothing is wrong with 
CLT. The point is that CLT says that p{x) converges to GD 
asymptotically at — > oo. On the other hand in any practical 
case is a finite quantity. At finite N the profile of prob- 
ability density for x may have nontrivial dependence on N . 
To stress this dependence p{x) at finite A'' will be denoted be- 
low as p„ {x). Then, if p^^ [x) has a non-Gaussian tail, which 
begins at |a;| 3> Xc{N) and if Xc{N) ^ oo at iV ^ oo, it 
does not contradict CLT. What I am going to show is that the 
occurrence of such a tail is a generic feature of the problem. 

To begin with let us consider two following examples 
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at ^ < -1/2 
fiO = <( 1 at - 1/2 < e < 1/2 
at C > 1/2 



/(O - A/\^r at Id » Cc, 
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where /(^) denotes the probability density for random ^ 
Eq. (jsj) ^, m and stand for certain positive constants. 

In case of Eq. % the conditions of CLT hold. However, 
it is evident that at any finite N p„ {x) vanishes identically 
at I a; I > A^/2 instead of exhibiting the exponential Gaussian 
decay at a; —> cxd [||]. In this case one has a superlight tail 
(actually, no tail at all). 

To see a non-Gaussian tail in case of Eq. (|]) let us first cal- 
culate the probability pi {x)dx that x has a certain large value 
from the interval xi < x < xi + dx because of contribu- 
tion of a single, anomalously large term, while all other terms 
in sum Eq. (^ are much smaller than this single term. The 
probability that a term in Eq. ([l|) has the value ^„ = x, where 
X belongs to the specified region is f{x)dx. The probability, 
that |^„| is much smaller then |x| is J^^f{C)dC, C < 
Due to independence of ^„ the probability that, say, S^i ~ x 
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and <^ \x\ at 2 < n < N is f{C)dC]^-^f{x)dx. 
Finally, bearing in mind that there are N terms in sum Eq. (|l]) 
and every one may be large, we obtain the following expres- 
sion for the total probabihty of the desired event 



pi{x)dx 
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Nf{x)dx : 

H N-l 



Nf{x)dx. 



Let us suppose that x is so large that the following conditions 
hold 

In this case the probability density pi{x) is reduced to the 
expression 



Piix)^Nf{x), 



(4) 



i.e., a heavy tail arises, see Eq. (|3|). 

Note, that according to CLT the probabihty density p{z) for 
the normalized sum 



z EE x/{aVN), 
where = (^^), should converge to GD at m > 3, viz. 

=Pg(^) = — ^"^xp ^-y^j at ^ oo. (5) 



Here and in what follows (. . .) denotes average over ^. 

GD Eq. ^ describes the probability density for the sum to 
have a value x due to contribution of a big number of individ- 
ual terms. Naturally, it cannot be smaller than the probabil- 
ity density related to contribution of a single anomalously big 
term. In other words GD Eq. (Q) certainly fails to describe the 
actual situation, when the condition p^ (z) > pi {x{z))dx/dz 
is violated [^. It provides us with the applicability condition 
for the GD \z\ <^ z^, where stands for the solution of the 
equation Pq{z) = pi{x{z))dx / dz. At large N the approxi- 
mate solution of this equation obtained by iterations is in the 
following 

Za ~ [(m- 3)ln7V+ 

mln((TO-3)lnA^ + ...)]^/^ > 1. 

Thus, for the problem in question the GD is invalid when the 
value of X becomes much greater than the standard deviation 
for the given GD. 

In what follows I am going to prove that the above result 
is a generic property of the problem, which is not connected 
with the particular type of /(^). Naturally, it is supposed that 
/(^) decays at big |^| sharp enough, so that the variance 
remains finite. Otherwise instead of GD one will obtain a 



Levy distribution (LD). The case of LD will be considered 
later separately. 

Let us calculate the characteristic function ^Pfj{Lo) = 
(exp(ici;z)), where w is a real quantity. By definition 



exp(iti;z)p„ (z)dz 



(6) 



On the other hand, employing Eq. (|l]) and independence of ^„ 
one can obtain 
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Eqs. (H) - (Q) supplemented with the inverse Fourier transform 
give rise to the following expression forp„ (z) 



Pjv(^) = ^ I e^p[-iujz + Nliig^{uj)]duj. (8) 



It should be stressed that Eqs. (p|) - are just a sequence of 
identities. Thus, expression Eq.T§) is an exact result, valid at 
any value of [@] 
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FIG. 1 . Contour of integration for Eq. ( |l l[ ) 

To obtain GD from Eqs. (Q) - let us consider the limit 
of large N and small lj. Expansion of the integrand in Eq. 
in powers of \/iV and integration (I remind the reader that 
(0 = 0) yield 



q.,[UJ) = 1 h o 



iVlng„H 



Then, inserting Eq. (llOh in Eq. (pi) one obtains 
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The integral is taken by integration in the plane of complex w 
along the contour shown in Fig. |l]. 

The integrand is an analytic function inside the contour, 
hence the integral over the entire contour is zero. The in- 
tegrals along the vertical segments vanish exponentially at 



Lo ±cx3. Therefore the integral Eq. ( Jl 1| ) equals the one 
along the line — oo — «z < < oo — iz. It eventually results 
in GD Eq. (|). 

However, here we are focusing on the applicability condi- 
tions of these transformations rather then on the well-known 
result of the integration. The transformation of the integral 
Eq. (^Tj) into the one in the complex plane implies that ex- 
pansion Eq. is now performed in the complex plane too. 
Thus, the applicability conditions for the expansion say that 
\lo\^ I'M ^ 1, where to = uj' ^ iuj" . This condition must 
hold at least for those cj's, which make the main contribution 
to the integral. For the integral along the line —oo — iz < 
u! < oo — iz the main contribution is made by uj' = 0(1). 
However, the imaginary part of these cj's is —z. We are in- 
terested in the tail of where \z\ ^ 1, so for ut with 
uj' = 0(1) ^ \z\ we have |a;| = \z\, which finally yields the 
following applicability condition for GD Eq. (pb 



< N 



(12) 



Generally speaking, it means that the tail of actual p„ (z) 
given by Eq. (|]) differs from that for Eq. (§) at z^ of order of 
(or grater than) N. The only exception from the rule is the 
case when 



9n{^) = exp ( 



(13) 



and expansion Eq. ( jlO|) t ransforms into an identity. Equaliz- 
ing Eq. (^ and Eq. (|13|) and employing the inverse Fourier 
transform, one can recover the only /(^), which brings about 
a Gaussian tail for Pj^{x). This /(^) is as follows 
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It is worth mentioning that contrary to all other cases, when 
GD is an asymptotic law valid at large N only, now Eq. (||) 
is the exact result, which is good at any N and any z. This 
property reflects the well known fact of self-similarity of a 
Gauss distribution |Q 

Thus, we have arrived at a number of important conclusions 

• A sum of any finite number of independently and iden- 
tically distributed random variables has a Gaussian tail 
if and only if the random variables themselves have a 
Gaussian distribution. 

• In any other situation the probability density p^^ (z) for 
the normalized sum z has a tail, which differs from the 
Gaussian. 

• Contrary to the GD for the normalized sum z the non- 
Gaussian tail is not universal. Its profile depends on the 



distribution of the random terms contributing to the sum 
and may be either heavier than the Gaussian, or lighter 
than it. 



• The non-Gaussian tail begins at \z\ of order of v ^ and 
lasts up to infinity. 

• The non-Gaussian tails exist at finite N only. It disap- 
pears in the limit N oo. 

To illustrate these general conclusions let us consider in de- 
tail several particular examples of /(^) and obtain the corre- 
sponding profiles of p„ (z) based upon Eqs. (^ - . In case 
of Eq. (0) simple calculations yield 
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so that 
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First I show that the above expression forp„ (z) does fulfill 
the condition p„ (z) = at |z| > \fN /{2a). To this end let 
us extend the integration in Eq. ( |l4| ) to the plane of complex 
u) and note that the integrand is an analytic function of lo at 
any finite |cj|. For this reason the corresponding integral over 
any closed contour in the complex plane is identical zero. For 
definiteness in what follows I suppose that z > (the case 
of negative z is analyzed analogously). Then, I consider a 
closed contour consisting of two elements — a segment of the 
real axis — i? < lu' < R and a circular arc with the radius R 
lying in the lower half-plane, which connects the edges of the 
segment. Next, note that the greatest term of the integrand in 
the lower half-plane has the form 



const 



N 



■ exp 




It is straightforwardly seen from the above expression that 
the integral over the arc tends to zero at z > ^/N /{2a) and 
i? — > cx). Finally, bearing in mind that the integral over the en- 
tire closed contour is zero, I conclude that the integral Eq. ( |l4| ) 
does turns into identical zero at z > /{2a). As an ex- 
ample the plot p„ (z), obtained by numerical integration of 
Eq. (14) at = 10 is presented in Fig. (^. 

Next, we discuss Pj^{z) at /(^) resulting in heavy tails. 
Specifically we consider the normalized probability density 
/(f) in the form 



/(O 



A 



1 + e 



21 ■ 



(15) 



where / is any positive integer number Thus, the case in 
question is a particular realization of /(f) given by Eq. (|^). 
Straightforward calculation yields the following expression 
for the variance all > 2 
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At Z = 1 the variance diverges. It is trivial to obtain p{z) 
at I = 1. The only modification to the described approach is 
that the normalized sum z in this case should be defined as 
x/N. For this z Eq. (0) yields g„(u;) = exp(— jwl/A^). Next, 
Eq. (M) brings about the final formula (a Levy distribution) 



1 1 



(16) 



TT 1 + z2 ■ 

It should be stressed (i) the expression Eq. ( p^ is an exact 
result, valid at any N, and (ii) the profile of p{z) in Eq. (|l6|) 
coincides with that for /(^) in Eq. ( p^ ) at Z = 1 — a Levy dis- 
tribution possesses the same self-similar properties as a Gaus- 
sian does [^] . However, here we are focused on the tail behav- 
ior rather then on the self-similarity of distributions. Follow- 
ing this line, I would like to emphasize that the tail of p„ (x), 
which Eq. ( p^ leads to, has the form of Eq. and obviously 
may be explained in the same terms P]. 




FIG. 2. Plots of Gaussian distribution Eq. (^) for the normalized 
sum z (thin line) and probability density p^^ (z) calculated numeri- 
cally according to Eq. ( p4| ) at A'^ — 10 (thick line) exhibiting a tail 
lighter than the Gaussian. Note, that in agreement with Eq. (112) de- 
viation of p„ (2) from the GD begins at z ~ \/iV. 

To calculate p„ (z) in case of arbitrary / < 2 let us employ 
the fact that (w) is a real even function of uj at any real 
even /(^), see Eq. (0), and hence for such /(^) Eq. (^ may 
be transformed as follows 
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'-dz. 
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Thus, it suffices to calculate (uj) for positive lo only. Next, 
extending integral Eq. (^ into the plane of complex ^, note 
that the for /(^) given by Eq. ( [l5| ) integrand has / simple poles 
in the upper half-plane. The poles are located in the points 



6 = exp 



Then, the integral is equal the following expression 

9. (-) - ^ (s- ^) E exp[Hl(2, + l)(2.-l)] 



(18) 



Calculation of p{z) according to Eqs. ([T^), (|l8]) is rather cum- 
bersome, see Appendix. It yields the following result 



Pn {z) = 



IN 



^T{a^/N) 



,21-1 



at z > 



N 



(19) 



Plots 



of Pj^{z) obtained by numerical integration of 
Eqs. (jnl), dll) at / = 2 and two values of iV (A^ = 10^, lO'^, 



respectively) are presented in Fig. |] The crossover from GD 
to heavy tails Eq. (|l9|) is seen clearly. In agreement with the 
above discussion the larger is N the longer the corresponding 
curve follows GD. 
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FIG. 3. Plots of Gaussian distribution Eq. (|5|) shown as a thin full 
line and probability densities p„ (z) calculated numerically accord- 
ing to Eqs. |l|) at Z = 2 for = 10^ (thick dashed) and 
A'^ — 10* (thick dot-dashed). 'Bothp^{z) exhibit tails /leai'/er than 
the Gaussian. The corresponding tail asymptotics, Eq. ( p^ are drown 
as thin dashed and dot-dashed lines respectively. 

Comparing Eq. ( p^ with Eq. (Q) one can see that the two 
expressions yield exactly the same asymptotic behavior for 
the probability densities [Q, including the values of the nor- 
malizing multiplier [see the definition of A in Eq. (|5])]. It 
should be stressed, however, that this is not the case for any 
/(^) with a tail lighter the the Gaussian. Eq. (^ describes the 
probability density for a sum to have a great value because of 
contribution of a single extremely great term. Meanwhile if 
/(^) decays sharp enough the probability of such an event is 
very small. In this case it could be more probable to obtain 
the great value of the sum due to contribution of several great 
but not extremely great terms. It means, that the actual tail 
of p„ (x) just cannot be lighter than that given by Eq. (Q) but, 
in principle, it could be heavier. To elucidate this issue let us 
consider the following profile of /(^) 



f{0 = 



1 



TT cosh ^ 



The corresponding variance cr^ equals 7r^/4. Function g„ (oj) 
is calculated according to Eq. (^ with standard methods of 
integration in a complex plane. The integration yields 
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Then, (z) is given by the following integral 

, , ctVN exp(-2zzC^)^^ 
J-oo cosh C 



(20) 



where C. = oj'k / {2a\/ N). To find the asymptotics of integral 
Eq. (|o|) at 3> iV let us extend the integration to the plane 
of complex C, and consider a standard contour, where the edges 
of the segment — i? < C < i? are connected through the lower 
half-plane by a circular arc with the radius R. The integrand 
has infinite number of poles of order N lying in points C," = 
(2n + l)7r/2 of the imaginary axis. Here n is any integer. 
Bearing in mind that the integral over the arc tends to zero at 
R ^ CO and the direction of circulation about the contour is 
negative one obtains 



(21) 
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exp(-2zzC^) . 2n+l 
cosh^ C 2 



In its turn 



res 



exp(-2izC^) 2n+l 
; — iTT — ^ — 



cosh^ C 



lim 
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-iN-l 



C_^-i^(2n+l)/2 \ {N - 1)! 

\ exp -2^zC 

cosn^ I \ TT 



(22) 



We are interested in the leading (in z 3> v^) term of 
Eq. (22). Note, that each differentiation of (. . contributes 
a multiplier of order of N, while each differentiation applied 
to Gxp(. . .) yields the one of order of z\^ ^ N. Thus, to 
obtain the leading term of Eq. ( ^2| ) one must apply all iV — 1 
derivatives to the exponent. It gives rise to the following ex- 
pression for the residue 



res 



exp{-2tzC^) 



2n+l 



cosh^ C 



(23) 
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(I remind to the reader that a = 7r/2). It should be empha- 
sized that the above expression gives rise to p,^{z), which 
is much greater than that following from Eq. (Q). A plot of 
p„{z) obtained by numerical integration of Eq. (pOl) as well 



as its comparison with GD and asymptotic Eq. ( |24| ) are shown 
in Fig. 




FIG. 4. Plots ofjprobability density p„ (z) calculated numerically 
according to Eq. (EQ) at TV = 25 (thick line), Gaussian distribution 
Eq. (^ for the normalized sum z (thin line) and asymptotics Eq. ( p4| ) 
shown as a dashed line. See the text for details. 

Thus, the above consideration has shown that the very tail 
of the probability density p^^ (x) for a sum a; of iV random 
terms ^„ at any finite N is defined by the tail of the probabil- 
ity density /(^). The tail is lighter than a Gaussian, if /(^) 
decays at large ^ faster than GD, and heavier than it in the op- 
posite case. The only situation, when the tail of p„ (x) is the 
Gaussian, corresponds to a Gaussian /(C). For /(C) decaying 
as a certain power of C large fluctuations of x occur because 
of contribution to the sum of a single anomalously great term, 
rather than due to the ones of a number of terms of moderate 
values Jiol], while for exponentially decaying /(C) this is not 
the case. 

I benefited much from discussions of this work with N. 
Makarenko and H. Takayasu. This study was supported by the 
Grant-in- Aid for Scientific Research (No. 11 837006) from the 
Ministry of Education, Culture, Sports, Science and Technol- 
ogy (Japan). 



where dots denote dropped higher order (in V N / z) terms. 

Substitution of Eq. ( |23| ) in Eq. ( |2l] ) transforms the sum 
there into a geometric progression, which may be summa- 
rized. However, the summarizing is not required — due to 
the exponential decay of the sum's terms the sum equals to 
its first term with the exponential accuracy. Finally the de- 
sired tail of the probability density at z 3> is given by the 
following formula 



(24) 



{N-iy. 



z exp 



APPENDIX: 



Here I discuss the derivation of Eq. (|19|). Note, that if uj is 
regarded as a complex variable, then (cj) given by Eq. 
is an analytical function of it at any finite |cj|. Since is a 



positive integer number the same is true for (g„(ti>))^. For 
this reason {Qm) exp(— ici;z)(iaj — 0, where C stands for 
any closed contour in the plane of complex lo. 

To obtain the asymptotic behavior of p{z) at large z let us 
consider contour C consisting of segments of real (0 < oj' < 
R and imaginary (0 > w" > —R) axes and a circular arc 
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with the radius R, connecting the edges of the segments. Ac- 
cording to Eq. ( |r^ ) the integrand decays along the real axis as 

In the lower half-plane it de- 



exp 



{uj'VN/a) sin(7r/20 



cays not slower than exp 



Then, it is 



easy to show that at z ^ \/N ja the integral over the circu- 
lar arc tends to zero at i? ^ oo. Thus, in the specified limit 
z ^ \/N I a one obtains 
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Next, note that cxp(— is a sharp function relative to 

(Sjv (~*^")) ■ III this ^^^^ fin'i leading asymptotic of 
at z ^ cx) one may expand (7„(— icj") in powers of 
uj/{a\/N), truncating the expansion when the first term with 
nontrivial contribution to PjJz) is obtained. 

The replacement in Eq. ( 1 8 ) w with — iuj" and the expansion 
of exponents in the numerators results in 
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The latter sum over j is a geometric progression. Summariz- 
ing it up one obtains 
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i7r(fc + 1) 
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2/ [l-exp(i7r^)] atfc 
at fc = 2p + 1 7^ 2g/ - 1 
ldA,k = 2ql-l 
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Herep = 0, 1, 2, . . . ; 9 = 1, 2, 3, , 



Then, Eq. (A2) may be transformed as follows 



p=0 



where 
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(A3) 



7r(2p+l) 
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^^-^''"'2^ {2ql-iy. \aVN 
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At small 07, retaining just the first terms in both the sums 
in Eq. (A3) one arrives at the following result gj^{~iuj") = 



1 + ... + isi + .., and next, Im (g^ (— iw"))^ = Nsi 
where dots denote dropped higher order in uj" / (ctVTV) terms. 
Substitution it into Eq. ( Al) and integration yield Eq. (p^. 



* E-mail: tribel@scroll.apphy.fukui-u.ac.jp 
[1] B. V. Gnedenko, and A. Kolmogorov, Limit Distribution 

for Sums of Independent Random Variables (Addison- Wesley, 

Cambridge, MA, 1954). 
[2] G. Boffetta, V. Carbone, P. Giuliani et al, Phys. Rev. Lett. 83, 

4662(1999). 

[3] D. Roberts, and D. L. Turcotte Fractals, 6, 351 (1998). 

[4] V. Plerou, P. Gopikrishnan, L. A. Nunes Amaral et al, Phys. 
Rev. E 60, 6519 (1999). 

[5] T. Bohr, M. H. Jensen, G. Paladin and A. Vulpiani, Dynamical 
Systems Approach to Turbulence (Cambridge University Press, 
Cambridge U.K. 1998). 

[6] Taking the opportunity, I would like to thank H. Takayasu, who 
kindly pointed me out this simple and beautiful example of a 
non-Gaussian tail. 

[7] For the correct comparison of the probability densities one 
should remember that change of a random variable results in 
the change of the corresponding probability density according 
to the formula p(z) = p{x[z)){dx / dz) . 

[8] Note, to provide statistical description for x one should consider 
an ensemble of sequences {^n}, l<n<A'^ with the same 
value of TV for any sequence in the ensemble. This approach 
does not depend on the particular A'^. Thus, not only large but 
also any positive integer values of are meaningful. 

[9] See, e.g., M. F. Shlesinger, G. M. Zaslavsky, and J. Klafter Na- 
ture 363, 31 (1993). 
[10] Strictly speaking the latter was proved just for f{f) decaying as 
even power 5. However, bearing in mind Eq. (BI) the extension 
of the conclusion to any power decay sounds quite reasonable. 



6 



